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# A nonprofit, nonpartisan oraanization 
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»n: To increase government effectiveness 
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Recent independent assessment found 



Over the past five years, the Coalition has 
successfully influenced legislative language, increased 
funding for evidence-based evaluations and programs, 
helped shape OMB's Program Assessment Rating Tool, 
and raised the level of debate in the policy process 
regarding standards of evidence. The Coalition has 
established a generally positive reputation as a 
rigorous, responsive, honest, and impartial advocate 
for evidence-based approaches, primarily at the 
federal level. " 
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Robert Boruch - Co-Chair, 
Campbell Collaboration 

Jonathan Crane - fmr Sr Policy 
Advisor to Vice President 

David Ellwood - Dean, Harvard's 
JFK School 

Judith Gueron - fmr President, 
MDRC 

Ron Haskins - fmr Sr Advisor to 
President for Welfare Policy 

Robert Hoyt - Founder, Jennison 
Associates 

Blair Hull - Founder, Hull Trading 
Co 

David Kessler - fmr FDA 

Commissioner 



of Advisors 



Jerry Lee - President, Jerry Lee 
Foundation 

Dan Levy - Faculty Chair of MPA 
Programs, Harvard's JFK School 

Diane Ravitch - fmr Asst 
Secretary of Education 

Howard Rolston - fmr Director of 
Research, HHS/ACF 

Isabel Sawhill - fmr Associate 
Director of OMB 

Martin Seligman - fmr President, 
American Psychological Assn 

Robert Solow - Nobel Laureate in 
Economics, MIT 

Nicholas Zill - Vice-President, 
Westat 
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Problem : Tens of billions are spent annually on 
development, yet little is known about "what works 



♦ That was central finding of the recent Evaluation Gap 
report , based on its comprehensive assessment of the 
evaluation literature for projects funded by UN 
agencies, multilateral development banks, and 
developing country governments 



# Earlier World Bank reviews reached similar 
conclusions. 



■Or 



Rigorous evaluations have identified 
interventions that are ineffective/harmful 



# Vouchers for disadvantaged workers, to subsidize their 



employment 

Well-conducted randomized trial found large negative 
effects on employment. 



# Increasing grassroots participation in monitoring road 
projects in Indonesia 

Well-conducted randomized trial found no effect on 
corruption/waste in project implementation. 























































































Riaorous evaluations have identified a few 
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# OMB-led government-wide evaluation initiative: 



New federal evidence-based initiatives 
we've helped inform and/or shape: 



$100 million in new funds for rigorous evaluations in 
FY11 (many likely to be randomized trials) 



Newly-enacted initiatives to expand implementation of 
evidence-based interventions: 



Evidence-Based Home Visitation - HHS ($1.5B/5yrs) 
Teen Pregnancy Prevention - HHS ($110M) 



Social Innovation Fund - CNCS ($50 million) 
Investing in Innovation Fund - Dept of Ed ($650M) 
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We believe many types 
research/evaluation are needed 



Implementation studies . 



Well-conducted comparison-group studies, and small 
randomized trials (RCTs) , to identify promising 
programs that merit more rigorous evaluation. 



We generally advocate large RCTs of programs at-scale 
only when program has been shown (i) well- 
implemented, and (ii) highly promising. 



But a Central Theme of Our Work, Consistent With 
A Recent National Academies Recommendation 



■Or 



...is that evidence of effectiveness generally 
"cannot be considered definitive" unless 
ultimately confirmed in well-conducted RCTs, 
even if based on the next strongest designs. 
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■Or 



Second-Best When Random 
Assignment Is Not Possible: 



<t> Observably-equivalent comparison-group study, 

The groups should be: 



- -: : - - 



1 Highly similar in key characteristics; 



2. Not formed through self-selection (or other methods 
likely to create differences in motivation etc) 



3. Preferably, chosen prospectively (i.e., before the 



intervention is administered). 



























































































Less Rigorous Study Designs Include: 
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► Outcome metrics (without ref< 
control or comparison group). 
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Too Often, Promising Findings in Non- 
Randomized Studies Are Not Confirmed in 
Subs equent, More Definitive RCTs 



<► In medicine : 50-80% of interventions found promising 
in phase II (nonrandomized studies or small efficacy 
trials) are found ineffective in phase III (sizable RCTs). 



In K-12 education : 9 of the 10 large RCTs of 
education strategies funded by Institute of Education 
Sciences 2003-2010 found weak or no positive effects 



& Similar pattern occurs in other areas (e.g., 
welfare/employment, crime prevention, etc.) 



Example - AIDS Vaccine Failure in Africa 



■Or 



#> Head of AIDS Vaccine Advocacy Coalition, 
commenting on the vaccine failure: 



" 777/5 is ... an important milestone in many 
respects. This is the way products get 
developed. Lots of things don't work, and 
we're on the road to finding something that 
does.' 



- Mitchell Warren, July 2008 



Impact of Career Academies on 
Completion of a Postsecondary Credential 
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Academy 
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| | Bachelor's degree or higher 

J Associate's Degree 



□ 



Career Non-randomized 
Academy Comparison 
Students Group* 



Certificate/license 



*The comparison group consists 
of similar students in similar 
schools nationwide. Their 
estimated rates of 
postsecondary completion are 
statistically adjusted to control 
for observed differences in their 
background characteristics 
versus the program group. 



Source: Data provided by 
James Kemple, MDRC Inc. 



Job Training Partnership Act: Impact on 
Earnings of Male Youth (Non-arrestees) 
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Mean Earnings, by Quarter: Male Youth Non-arrestees 
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How might one rigorously evaluate a 
public diplomacy strategy? Example: 



# To be evaluated : A strategy to undermine public 
support for terrorist ideology in a country where such 
ideology is widespread (e.g., fund schools run by 
moderates, conduct media campaign on harm 
terrorism causes innocent civilians. 



# First : Identify ~60 communities where support for 
violence against Americans exists. 



Second : Randomly assign half to receive the strategy, 
and half to a control group. 



# Third: Measure attitudes towards terrorism vs. U.S. in 



treatment & control communities over time. 
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